Papers by Yang Trista Cao
Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models encode correlations between social groups and traits, like associating the group with the group. |
| Approach: | They adapt the Agency-Belief-Communion (ABC) stereotype model to a language model and introduce the sensitivity test (SeT) to measure stereotypical associations. |
| Outcome: | The proposed framework is used to measure stereotyping of intersectional identities in language models. |
Analyzing Stereotypes in Generative Text Inference Tasks (2021.findings-acl)
Copied to clipboard
| Challenge: | Social psychology studies how social stereotypes are shared as part of cultural knowledge . |
| Approach: | They study how stereotypes manifest when potential targets are situated in neutral contexts . they collect human judgments on the presence of stereotypes in generated inferences based on annotator positionality . |
| Outcome: | The results show that the annotators' positions differ depending on the type of inferences they generate . |
What’s Different between Visual Question Answering for Machine “Understanding” Versus for Accessibility? (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing benchmarking datasets for visual question answering focus on machine "understanding" but it remains unclear how progress on those datasets corresponds to improvements in this real-world use case. |
| Approach: | They evaluate the visual question answering task by evaluating a variety of VQA models. |
| Outcome: | The proposed model can achieve high scores on tasks thought to require human-like comprehension, including image tagging and captioning. |
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations (2022.acl-short)
Copied to clipboard
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, Aram Galstyan
| Challenge: | Recent natural language processing systems use large language models as the backbone . however, societal biases are encoded in these models and transferred to downstream applications . |
| Approach: | They propose to use two categories to measure fairness in natural language processing tasks . they find intrinsic and extrinsic metrics do not correlate in their original setting . |
| Outcome: | The proposed metrics do not correlate in their original setting, the authors show . they find that they are not accurate when correcting for metric misalignments and noise . |
Toward Gender-Inclusive Coreference Resolution (2020.acl-main)
Copied to clipboard
| Challenge: | a recent study shows that coreference resolution systems can be harmful to binary and non-binary trans and cis stakeholders. |
| Approach: | They propose to use gender-based crowd annotations to investigate coreference resolution biases . they use a dataset to examine the complexity of gender in crowd annotation systems . |
| Outcome: | a new study shows that without acknowledging and building systems that recognize gender, we build systems that lead to many potential harms. |